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1.  Introduction 


Classification  using  the  MetalMapper  advanced 
electromagnetic  sensor  was  demonstrated  in  conjunction 
with  a  Remedial  Investigation/Feasibility  Study  (RI/FS)  at 
the  Rocket  Pond  Range,  Ft.  Sill,  OK  in  late  2011.  This 
report  summarizes  the  results  of  that  demonstration.  The 
document  Implementing  Classification  on  Munitions  Response 
Sites  (Ref.  1)  provides  practical  information  for  deciding 
whether  classification  is  appropriate  to  a  particular  site  and 
how  it  is  best  implemented. 

Classification  is  motivated  by  the  need  to  perform 
munitions  response  more  cost-effectively  so  that  Umited 
clean  up  dollars  can  be  used  to  reduce  real  risk  on 
munitions-contaminated  sites  sooner.  The  estimated 
HabiUty  in  the  FYIO  Defense  Environmental  Programs 
Report  to  Congress  for  Munitions  Response  is  $15.2B. 

(Ref.  2)  The  bulk  of  this  liability  is  $10. OB  for  the  1703 
sites  identified  in  the  Formerly  Used  Defense  Sites  (FUDS) 
program  and  $4.4B  for  the  2433  sites  identified  on  Active 
Installations.  The  remaining  $0.8B  is  in  Base  Realignment 
and  Closure  (BRAC).  The  estimated  completion  dates  for 
many  sites,  particularly  in  the  FUDS  program,  are  decades 
out  if  they  are  to  be  cleaned  up  at  planned  funding  levels 
using  current  practice. 

When  a  munitions  response  site  is  cleaned  up,  in  most  cases,  it  is  mapped  with  a  geophysical  sensor 
and  the  locations  of  aU  detectable  signals  are  excavated.  Geophysical  sensors  detect  metal  and, 
therefore,  many  of  the  detections  do  not  correspond  to  munitions,  but  rather  to  harmless  metallic 
objects.  Field  experience  indicates  that  95-99%  or  more  of  objects  are  found  to  be  nonhazardous. 
Current  standard  practice  does  not  provide  a  means  to  discriminate  between  munitions  and  other 
items,  termed  ''clutter.”  As  a  result,  most  of  the  costs  to  remediate  a  munitions-contaminated  site 
using  current  methods  are  spent  on  excavating  targets  that  pose  no  threat. 

Classification  is  a  process  used  to  make  a  decision  about  the  likely  origin  of  a  signal.  In  the  case  of 
munitions  response,  high-quaHty  geophysical  data  can  be  interpreted  with  physics-based  models  to 
estimate  parameters  that  are  related  to  the  physical  attributes  of  the  object  that  resulted  in  the  signal, 
such  as  its  physical  size  and  aspect  ratio.  The  values  of  these  parameters  may  then  be  used  to 
determine  whether  the  signal  arose  from  a  munition  or  harmless  clutter.  With  reliable  classification, 
only  the  munitions  need  to  be  removed  from  the  site. 

The  Strategic  Environmental  Research  and  Development  Program  (SERDP)  and  the  Environmental 
Security  Technology  Certification  Program  (ESTCP)  have  supported  the  development  of  purpose- 


KocketPondKange^  Ft.  Silly  OK  — 
ongoing  Rlj  FSy  relatively  flat  and  openy 
moderate  geologic  interference 

Munitions  —  wide  variety  of  munitions 
from  40 -mm  grenades  through  medium 
caliber  projectiles  and  a  variety  of 
rockets 

Results  —  MetalMapper  was  used  to 
successfully  classifif  greater  than  99%  of 
the  targets  of  interest  and  eliminate 
about  75%  of  the  clutter.  M  production 
contractor  field  crew  collected  high 
quality  cued  MetalMapper  data.  Roth 
geophysicists  from  the  production 
contractor  and  the  developers  of 
classification  methods  were  successful  in 
using  these  data  to  achieve  substantial 
classification. 
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built  advanced  electromagnetic  sensors  and  associated  analysis  methods  for  classification.  Following 
the  successful  demonstration  of  classification  methods  in  controlled  test  environments,  ESTCP 
initiated  a  Classification  Pilot  Program  to  validate  the  application  in  real-world  conditions.  The  goal 
of  the  program  is  to  demonstrate  that  classification  decisions  can  be  made  using  an  explicit 
approach,  based  on  principled  analysis  that  is  transparent  and  reproducible.  The  demonstrations  are 
planned  and  conducted  in  cooperation  with  regulators  and  program  managers  in  the  Services. 

The  physics  governing  the  electromagnetic  response  of  a  metal  object  is  well  understood  and 
predictable.  Data  collected  with  these  sensors  contain  the  same  information  content  on  any  site  and 
demonstrations  to  date  have  confirmed  that  classification  works  predictably.  Nevertheless, 
demonstrations  will  be  required  at  a  number  of  sites  to  represent  the  wide  variability  in  munitions 
types,  target  densities,  terrain,  vegetation,  geology,  land  use  history,  future  land  use,  and  other  site 
characteristics  that  will  affect  the  applicability  of  classification  and  to  establish  cost  effectiveness  and 
implementability.  The  demonstrations  also  present  an  opportunity  to  work  out  standard  operating 
procedures  and  establish  quality  control  (QC)  measures.  Prior  demonstrations  have  been  conducted 
at  a  number  of  sites  across  the  country.  Details  about  past  and  ongoing  demonstrations  can  be 
found  on  the  SERDP-ESTCP  web  site  at  http:/ / serdp-estcp.org/Featured-Initiatives/Munitions- 
Response-Initiatives /Classification- Applied-to-Munitions-Response. 

The  demonstration  at  Ft.  Sill  continues  the  practice  of  production  geophysics  contractors  collecting 
and  analyzing  advanced  sensor  data  using  the  MetalMapper.  One  purpose  of  the  demonstration  was 
to  train  production  contractors  in  the  analysis  of  data  from  these  advanced  sensors.  This  is  an 
important  consideration  in  evaluating  and  applying  the  results.  We  discourage  potential  customers 
from  using  the  results  of  any  single  demonstration  to  rank  performers  and  make  contracting 
selections;  analysts  will  gain  experience  and  improve.  Data  were  also  analyzed  by  experienced  teams 
from  the  developers  of  the  classification  methods.  Table  1  shows  the  participants  and  their  roles  in 
the  demonstration. 

Table  1.  Participants  in  the  MetalMapper  Demonstration  at  Ft  Sill 


Task 

Performer(s) 

Task 

Performer(s) 

Site  Preparation 

Parsons 

MetalMapper  Data 
Analysis 

Dartmouth  College 

Parsons 

SAIC 

US  Army  Corps  of 
Engineers 

EM61-Mk2  Data 
Collection  and  Target 
Selection 

Parsons  as  part  of  the 
ongoing  RI/FS 

MetalMapper  Data 
Collection 

Parsons 

Intrusive  Investigation 

Parsons 

Scoring 

Institute  for  Defense 
Analyses 
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2.  Ft.  Sill  Demonstration  Flow 

The  sequence  of  the  demonstration  is  outlined  in  the  flow  chart  in  Figure  2-1. 


Figure  2-1.  Flow  chart  outlining  steps  in  the  demonstration  at  Ft  Sill.  Green  boxes  are  tasks 
performed  as  part  of  the  RI/FS,  blue  boxes  are  tasks  performed  by  ESTCP.  Others  are  tasks 

performed  by  contractors. 

Prior  to  the  beginning  of  data  collection,  an  instrument 
verification  strip  (IVS)  was  installed  and  the  site  was  seeded 
with  inert  munitions  and  small  industry  standard  objects 
(ISOs),  1-in  nominal  X  4-in  pipe  nipples.  (Ref.  3)  Data 
collectors  visited  the  IVS  twice  daily  to  verify  equipment 
function  at  the  start  and  end  of  each  day.  Since  there  are 
few  native  unexploded  ordnance  (UXO)  on  any  munitions 
response  site,  the  seeds  provided  sufficient  targets  of 
interest  (TOI)  to  allow  a  statistically  defensible 
determination  of  the  correct  classification  of  TOI. 

As  part  of  the  RI/FS,  the  site  was  surveyed  with  an  EM61 
array  to  provide  an  initial  Hst  of  detected  anomalies.  The 
MetalMapper  was  used  to  collect  cued  data  over  each 


Targets  of  Interest  (TOI)  are  all  objects 
that  must  be  removed  from  the  site. 
Typically  the  TOI  will  include  all 
known  or  suspected  munitions  types,  any 
other  unexpected  munitions,  munitions 
parts  such  as  fuyes  that  present  an 
explosive  hayard,  and  all  seeded  items. 
When  classification  is  applied  to  a  site, 
the  local project  team  will  decide  what 
items  constitute  TOI. 
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anomaly  in  the  grids  selected  for  this  demonstration.  All  detected  targets  were  dug  up  to  provide 
complete  ground  truth  for  the  purposes  of  determining  performance.  The  UXO  technicians 
photographed  each  item  that  was  dug  and  recorded  its  location,  depth,  and  description. 

The  geophysical  data  were  passed  to  the  data  analysis  teams.  A  complete  overview  of  the  analysis 
procedures  can  be  found  in  Ref  1.  Briefly,  the  analysts  used  methods  based  on  the  dipole  model  to 
estimate  target  parameters.  Analysts  were  offered  training  data  from  test  pit  measurements  and  the 
opportunity  to  request  additional  training  data  from  the  recovered  targets,  as  though  they  were 
doing  a  limited  number  of  sample  digs.  These  data  were  used  to  set  classifler  rules  —  the  decisions 
that  separate  the  anomalies  into  TOI  and  non-TOI.  The  classiflers  were  then  applied  to  all  of  the 
targets  that  remained  blind  for  each  demonstrator.  Since  training  data  was  by  request,  the  blind 
target  set  was  different  for  each  demonstration. 

The  product  required  from  each  analyst  was  a  ranked  anomaly  list  as  shown  in  Figure  2-2.  One  and 
only  one  judgment  was  required  for  each  entry  on  the  anomaly  list.  Following  any  classifler  training 
data,  the  first  items  on  each  anomaly  list  are  those  targets  for  which  reliable  parameters  cannot  be 
extracted  and  therefore  must  be  dug.  Next  are  those  items  which  the  analyst  is  the  most  confident 
are  TOL  The  items  are  ranked  according  to  decreasing  likelihood  that  the  item  is  a  TOL  Any  items 
which  the  analyst  was  able  to  analyze  but  was  not  able  to  make  a  classification  decision  on  at  this 
time  were  placed  next  on  the  anomaly  list.  Last  are  all  those  items  that  the  analyst  was  confident  are 
not  TOI  ranked  by  their  likelihood.  This  initial  Ust  is  shown  in  the  left  panel  of  Figure  2-2. 


Initial  Ranked  Anomaly  List 


Rank 

Anomaly 

ID 

Decision 

Statisitic 

Dig 

Comment 

2498 

- 

Y 

Classifier  training  data 

1 

247 

- 

Y 

Can’t  extract  reliable  parameters 

5- 

313 

Y 

First 

Pass 

Threshold 

704 

N 

Unable  to  classify 

863 

N 

N 

N 

High  likelihood  not  TOI 

N 

N 

N 

.02 

N 

.02 

N 

N 

.01 

N 

Final  Ranked  Anomaly  List 


Figure  2-2.  Initial  and  Final  Ranked  Anomaly  Lists.  A  detailed  description  is  in  the  text. 

The  seeds  were  divided  into  QC  seeds  and  bUnd  seeds.  When  analysts  submitted  their  initial 
prioritized  lists,  the  QC  seeds  were  used  to  provide  feedback  if  seed  targets  were  missed.  Analysts 
were  also  provided  with  the  ground  truth  information  on  all  anomalies  in  the  red  part  of  their  lists 
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and  any  requested  anomalies  in  the  yellow  part.  This  is  signified  by  the  threshold  on  the  left  side  of 
Figure  2-2.  Based  on  this  information,  the  analysts  were  then  allowed  to  revisit  their  rankings  and 
assignments  for  all  items  that  were  still  blind  until  they  were  satisfied  that  the  best  possible 
classification  had  been  achieved. 

In  the  final  Hst,  shown  in  the  right  panel  of  Figure  2-2,  the  analyst  was  required  to  provide  a 
threshold  that  corresponds  to  the  division  between  those  items  recommend  for  digging  and  those 
that  can  safely  remain  in  the  ground.  That  is,  the  Hst  is  all  red  and  green  with  a  threshold  separating 
the  two  categories.  The  final  prioritized  anomaly  lists  were  scored  against  the  emplaced  bHnd  seeds 
and  recovered  targets  by  IDA. 

3.  Site  Description  and  Preparation 

Fort  Sill  is  located  in  Comanche  County  in  southwestern  Oklahoma,  adjacent  to  the  City  of  Lawton, 
Oklahoma,  and  occupies  approximately  94,000  acres.  (Ref.  4)  The  Wichita  Mountains  National 
WildHfe  Refuge  is  located  along  the  northwest  boundary  of  the  installation.  Oklahoma  City  is 
located  about  90  miles  northeast  of  Fort  Sill  and  Dallas-Fort  Worth  is  approximately  180  miles 
south. 

Fort  Sill  was  originally  called  Camp  Wichita  and  was  created  in  1869  by  Major  General  PhiHp  H. 
Sheridan  who  led  a  campaign  into  Indian  Territory  to  stop  hostile  tribes  from  raiding  border 
settlements  in  Texas  and  Kansas.  MG  Sheridan  later  renamed  it  in  honor  of  Brigadier  General 
Joshua  W.  Sill,  killed  during  the  Civil  War.  In  1901,  the  City  of  Lawton  was  estabHshed  along  the 
southern  boundary  of  Fort  Sill.  The  Fort  Sill  mission  has  changed  from  cavalry  to  field  artillery  and 
air  defense  artillery.  The  Rocket  Pond  Area  (Figure  3-1)  was  used  for  weapons  systems  training 
involving  40mm  grenades,  66-mm  Hght  anti-tank  weapon  (LAW)  rockets,  and  2.36-in  and  3.5-in 
rockets. 

A  UXO  clearance  was  conducted  on  the  area  in  1994.  (Ref.  5)  The  range  was  surface  cleared  over 
all  grids  (approximately  400  acres).  The  pond  basin,  dam,  and  other  areas  (approximately  194  acres) 
were  subsurface  cleared  to  a  depth  of  one  foot.  During  the  clearance,  287  UXO  items  were 
destroyed  out  of  115,500  anomaHes  excavated  (0.25%),  133,000  items  related  to  UXO  were  removed 
and  190,980  pounds  of  scrap  were  removed. 

Fort  Sill  is  scheduled  for  mission  growth  due  to  BRAC  which  has  resulted  in  the  need  for 
infrastructure  upgrades.  An  RI/FS  is  being  conducted  on  the  Rocket  Pond  Area  (—555  acres)  to 
faciHtate  anticipated  construction  at  this  site.  At  the  same  time,  a  MFC  clearance  project  (156.91 
acres)  is  being  conducted  along  the  52'''^  Street  right  of  way  adjacent  to  the  RI/FS  area  to  support 
base  upgrades.  The  munitions  discovered  during  this  clearance  project  served  as  the  source  of  QC 
seeds  for  this  demonstration  as  well  as  valuable  guides  for  the  data  analysts. 
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Figure  3-1.  Aerial  photograph  showing  Rocket  Pond  RI/FS  area  outlined  in  blue.  The  52*^^^  Street 
clearance  project  was  conducted  along  the  road  running  along  the  western  and  northern  boundaries. 


The  suspected  munitions  in  this  demonstration  area  include,  but  are  not  limited  to: 

•  40-mm  HE  and  practice  grenades, 

•  MKII  hand  grenades, 

•  M9  anti-tank  (AT)  rifle  grenades, 

•  Ml  1 A2  practice  AT  rifle  grenades, 

•  2.36-in  and  3.5-in  high  explosive  anti-tank  (HEAT)  rockets, 

•  66-mm  LAW  rockets, 

•  Signal/Illumination  flares, 

•  fuzes  (grenade,  projectile,  and  rocket),  and 

•  Hmited  evidence  of  20-mm  and  37-mm  projectiles. 


The  objective  of  the  demonstration  was  to  detect  and  correctly  classify  all  TOI  37mm  and  larger. 
The  analysts  were  provided  information  about  the  historical  use  and  known  munitions  types.  But, 
the  direction  specifled  that,  in  addition  to  these  munitions,  any  unexpected  munitions  would  also  be 
considered  TOI. 

At  a  live  site,  the  number  of  UXO  is  usually  small,  far  from  enough  to  provide  adequate 
opportunities  to  demonstrate  that  classiflcation  can  reliably  identify  the  TOI.  This  may  not  be  true 
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on  a  heavily-used  training  range  but  we  have  no  way  of  knowing  this  in  advance.  To  be  safe,  the  site 
was  seeded  with  inert  TOI  to  serve  as  process  QC  checks;  because  of  this,  the  seeds  are  emplaced  at 
depths  which  should  lead  to  good  signal- to -noise  ratios.  The  seeds  are  listed  in  Table  3-1.  The 
seeds  included  inert  munitions  from  other  sites,  industry  standard  objects,  (Ref.  3)  and  a  small 
number  of  items  recovered  from  the  clearance  project  adjacent  to  the  demonstration  site.  The  ISOs 
are  also  considered  TOI  and  expected  to  be  both  detected  and  correctly  classified. 


Table  3-1.  Seeds  emplaced  for  the  Ft  Sill  demonstration 


Item 

Number 

Depth  (cm) 

Item 

Number 

Depth  (cm) 

Small  Industry  Standard 
Object 

34 

15-28 

Practice  hand 
grenade 

2 

12-15 

37-mm  projectile 

62 

10-29 

2. 3 6 -in  rocket 

3 

15-20 

40-mm  projectile 

22 

10-25 

LAW  rocket  motor 

1 

15 

Mkll  hand  grenade 

20 

10-20 

flare  canister 

2 

15 

rifle  grenade 

5 

10-19 

signal  flare 

1 

12 

No  attempt  was  made  to  separate  the  seeds  from  the  surrounding  clutter.  For  safety,  seeds  were 
emplaced  using  standard  anomaly  avoidance  procedures.  For  realism,  the  emplacement  teams  were 
instructed  to  replace  any  metal  dug  up  during  emplacement  back  in  the  hole  with  the  seeded  object. 

4.  EM61  Detection  Survey 

The  EM61-MK2  data  used  for  anomaly  identification  was  collected  as  part  of  the  RI/FS.  Before 
the  start  of  digital  geophysical  mapping,  vegetation  was  removed  as  needed  and  surface  debris 
removal  was  conducted  over  ~250  acres  of  the  site  guided  by  the  results  of  the  1994  UXO 
clearance.  The  site  was  then  divided  into  grids  to  facilitate  data  collection. 

Either  a  towed  array  of  three  EM61-MK2  instruments  arranged  side-by-side  or  a  single  sensor 
mounted  on  its  standard  wheels  was  used  for  the  detection  survey  depending  on  local  conditions. 

As  expected,  most  of  the  site  was  surveyed  with  the  towed  array.  A  response  threshold  of  10  mV 
for  the  sum  of  EM61-MK2  gates  1  through  3  was  used  for  anomaly  selection.  (Ref.  4) 

An  example  of  the  implications  of  this  threshold  choice  is  shown  in  Figure  4-1.  The  response  of  a 
single  EM61-MK2  (the  worst  case)  to  a  small  ISO  in  its  least  and  most  favorable  orientations  is 
plotted  as  a  function  of  depth.  The  10  mV  threshold  in  the  sum  channel  corresponds  to  signal 
amplitude  expected  for  a  small  ISO  in  it  least  favorable  orientation  at  30  cm  below  the  ground. 
Under  the  center  sensor  of  the  three  sensor  array,  the  same  item  at  the  same  depth  would  produce  a 
sum  channel  signal  of  12.4  mV. 

The  EM61  survey  of  the  Rocket  Pond  area  resulted  in  far  more  anomalies  than  could  be  could  be 
investigated  further  in  a  demonstration  of  this  scale.  Accordingly,  five  of  the  original  grids  were 
selected  for  the  MetalMapper  data  collection  and  analysis.  Figure  4-2. 
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Figure  4-1.  Sum  of  EM61-MK2  response  in  Gates  1  through  3  versus  depth  for  a  small  ISO 
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Figure  4-2.  The  five  grids  selected  for  the  MetalMapper  demonstration 


The  five  demonstration  grids  are  shown  in  more  detail  in  Figure  4-3.  They  were  chosen  to  represent 
the  range  of  anomaly  densities  found  at  this  site.  Grid  C09,  in  the  lower  right  of  figures  4-2  and  4-3 
is  far  from  any  target  area  and  likely  represents  the  site  background  away  from  any  target.  Grids 


3,833,000  3,833,500  3,834,000  3,834,500 


Grid  L04 


Grid  K04 


0  100  200  400  Feet 


•  ■  ■  ’  . 

,  r  ■, ,  • 

-  ,  4i”  ■  '  ^ 

•  P.  r’  ' 

Grid  I03 


Grid  C09 


Figure  4-3.  Detail  of  the  five  grids  used  for  this  demonstration 


K04  and  L04  are  adjacent  to  the  most  densely  contaminated  target  area  and  represent  a  test  of  the 
high-density  capabilities  of  classification.  The  final  two  grids,  101  and  103,  represent  intermediate 
densities  that  should  be  amenable  to  classification. 


A  total  of  1908  anomalies  were  selected  for  study,  distributed  as  shown  in  Table  4-1.  All  anomalies 
in  grid  C09  were  investigated  and  a  subset  of  the  available  anomalies  in  the  other  four  grids.  These 
anomalies  were  chosen  from  both  the  highest  density  area  in  each  grid  and  an  area  of  intermediate 
density. 


Table  4-1.  Anomalies  investigated  in  each  grid 


Gtid 

Number  of  Anomalies 

C09 

126 

101 

430 

102 

445 

K04 

408 

L04 

499 

Total 

1908 

5.  MetalMapper  and  Data  Collection 

The  MetalMapper  developed  by  Geometries  is  designed  to  be  a  stand-alone  survey  and  cued 
detection  system.  The  system,  shown  in  Figure  5-1,  is  composed  of  three  orthogonal  1-m  x  1-m 
transmitters  for  target  illumination  and  7  three-axis  receivers  for  recording  the  response.  Its 
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sampling  is  electronically  programmable  and  therefore  flexible.  It  measured  the  decay  curve  up  to 
8  ms  after  the  transmitters  were  turned  off.  Centimeter-level  GPS  is  used  for  navigation  and 
geolocation  and  an  inertial  measurement  unit  (IMU)  is  used  to  measure  platform  orientation.  In 
cued  mode,  MetalMapper  is  positioned  over  each  anomaly  on  its  target  Ust  and  collects  the  full  suite 
of  data  while  stationary.  The  digital  data  set  produced  by  MetalMapper  is  fuUy  described  in  Ref.  7. 


Figure  5-1.  Schematic  and  photo  of  the  MetalMapper  as  used  at  Ft.  Sill 

In  this  demonstration  MetalMapper  was  used  only  in  cued  mode.  It  was  used  in  a  sled  conflguration 
mounted  to  the  rear  three-point  hitch  of  a  tractor.  A  commercial  geophysics  vendor,  Parsons, 
collected  MetalMapper  data  at  Ft.  Sill. 

Details  on  the  data  collection  and  QC  procedures  followed  by  Parsons  can  be  found  in  their 
demonstration  report.  (Ref.  8)  The  most  common  QC  failure  was  that  the  MetalMapper  was 
positioned  too  far  from  the  anomaly  to  obtain  reliable  parameter  estimates  because  of  inaccurate 
cueing  from  the  EM61  detection  survey.  If  the  separation  between  the  center  of  the  MetalMapper 
and  the  anomaly  location  was  more  than  40  cm,  the  anomaly  was  revisited  and  additional  data 
collected  within  the  40-cm  speciflcation. 

The  production  rate  for  the  cued  data  collection  was  117  targets  per  day.  This  is  lower  than  the  rate 
achieved  at  most  sites  and  was  primarily  due  to  intermittent  computer  failures  in  the  MetalMapper 
system.  Parsons  reported  an  overall  recollection  rate  of  5.8%. 

6.  Intrusive  Results 

The  intrusive  investigation  at  this  site  was  not  ideal  from  the  standpoint  of  a  classiflcation 
demonstration.  The  intrusive  crew,  who  were  working  in  support  of  the  RI/FS,  was  directed  to  the 
location  of  the  EM61-MK2  anomalies  rather  than  where  the  MetalMapper  collected  cued  data.  The 
MetalMapper  was  initially  positioned  at  the  exact  location  of  the  EM61  anomaly  but  in  many  cases 
the  MetalMapper  detected  no  item  directly  below  this  initial  position  and  so  was  moved  to  be 
centered  over  the  nearest  item  before  collecting  data.  Seventy  nine  anomalies  were  removed  from 
the  anomaly  list  because  the  item  recovered  was  more  than  0.6  m  from  the  position  of  the 
MetalMapper.  A  summary  of  the  items  recovered  at  the  remaining  1908  anomaly  locations  is  given 
in  Table  6-1. 
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Table  6-1.  Intrusive  Results  at  Ft.  Sill 


Recovered  Item 

Number 

TOI 

290  (150  seeds) 

Munitions  Debris 

1172 

Range-related  Debris 

95 

Cultural  Debris 

282 

No  Recovered  Item 

69 

Total 

1908 

A  further  breakdown  of  the  identities  of  the  TOI  recovered  is  given  in  Table  6-2.  The  ISOs  and  37- 
mm  projectiles  were  inert  seeds  as  were  all  but  one  of  the  40-mm  and  most  of  the  miscellaneous 
items.  Most  of  the  native  TOI  recovered  were  3.5-in  rockets  and  rocket  motors. 


Table  6-2.  TOI  Recovered  at  Ft.  Sill 


Description 

Number 

Description 

Number 

small  ISO 

38 

40-mm  grenade 

23 

37-mm  projectile 

62 

40-mm  frag  ball 

6 

3.5-in  rocket  motor 

122 

rifle  grenade 

5 

3. 5 -in  rocket 

7 

Mkll  grenade 

18 

2.36-in  rocket 

3 

Miscellaneous  seeds 

4 

7.  Classification  Results 

The  MetalMapper  data  were  analyzed  by  multiple  analysts,  including  both  the  developers  of  the 
analysis  methods  and  the  production  geophysics  crew.  Highlights  of  these  analyses  are  presented  in 
this  section. 

Figure  7-1  shows  the  receiver  operating  characteristic  curve  achieved  by  an  analyst  from  Parsons. 
(Ref.  8)  The  colors  on  the  plot  correspond  to  the  red  and  green  colors  in  the  final  ranked  anomaly 
Hst  as  shown  in  Figure  2-2.  The  red  are  the  items  the  analyst  classified  as  ''high  likelihood  TOT’  and 
the  green  are  those  the  analyst  called  "high  likelihood  not  TOL”  The  graph  plots  the  percent  of  the 
targets  of  interest  correctly  classified  on  the  vertical  axis  and  the  number  of  clutter  items  on  the 
horizontal  axis.  The  offset  from  zero  in  the  starting  point  reflects  the  small  amount  of  training  data 
that  the  analyst  requested  and  an  initial  black  line  would  represent  any  items  in  the  "unable  to  extract 
reliable  parameters”  category;  this  analyst  was  able  to  analyze  all  data.  The  blue  dot  represents  the 
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Figure  7-1.  Results  of  Parsons  analysis  of  the  MetalMapper  data 


threshold  selected  by  the  analyst  and  the  orange  dot  shows  the  point  on  the  ranked  anomaly  list 
where  100%  of  the  target  of  interest  are  captured.  Ideally,  a  classifier  would  correctly  identify  all 
targets  of  interest  in  the  red  with  zero  clutter  and  all  of  the  clutter  would  be  in  the  green.  In  this 
case,  the  red  part  of  the  curve  would  go  straight  up  to  100%  and  the  green  part  of  the  curve  would 
run  straight  across  the  top  axis.  Success  in  these  demonstrations  was  defined  by  eliminating  the 
maximum  amount  of  clutter  while  correctly  identifying  all  of  the  TOI.  Finally,  black  triangles  mark 
the  position  of  TOI  that  were  not  classified  correctly  by  the  analyst. 

In  this  demonstration,  there  were  1610  total  clutter  items  as  determined  from  the  ground  truth. 

This  analyst  was  able  to  correctly  identify  more  than  60%  of  them  at  his  threshold.  However,  seven 
TOI  were  incorrectly  classified  at  the  analyst  threshold.  It  would  have  required  over  200  more  digs 
for  this  analyst  to  have  correctly  identified  all  the  TOI. 

Figure  7-2  shows  the  results  of  the  SAIC  analysis  of  the  MetalMapper  data.  (Ref.  9)  This  analyst  was 
able  to  correctly  classify  over  80%  of  the  clutter  at  threshold  but  again,  seven  TOI  were  missed. 

This  analyst  would  have  had  to  dig  almost  the  entire  site  to  capture  these  TOI. 

The  TOI  incorrectly  classified  by  these  two  analysts  are  compared  in  Table  7-1.  The  two  Usts 
comprise  the  same  seven  anomalies  with  the  order  being  very  similar  between  the  two  analysts.  The 
difference  between  the  two  ROC  curves  results  from  the  Parsons  analyst  placing  these  items  in  the 
middle  of  the  ranked  anomaly  list  and  the  SAIC  analysts  placing  them  near  the  bottom  of  the  list. 
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Figure  7-2.  Results  of  the  SAIC  analysis 
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Table  7-1.  TOI  incorrectly  classified  by  two  analysts 


Parsons 

SAIC 

FS-1658 

FS-1658 

FS-1656 

FS-1656 

FS-1895 

FS-1488 

FS-1488 

FS-1895 

FS-1378 

FS-1378 

FS-1974 

FS-1579 

FS-1579 

FS-1974 

Six  of  these  common  misclassified  TOI  are  pictured  in  Figure  7-3.  Each  of  these  items  was 
identified  as  a  ‘'40mm  Frag  Ball  Cups  Containing  HE”  by  the  intrusive  team.  AU  six  of  these  items 
were  recovered  from  Grid  1-03,  adjacent  to  one  of  the  heavily-used  targets.  Neither  of  the  analysts 
had  ever  encountered  this  item  before  nor  did  they  know  to  expect  it.  This  illustrates  that  even 
experienced  analysts  may  miss  TOI  they  do  not  expect  to  encounter. 

Finally,  Figure  7-4  shows  the  results  of  the  analysis  by  researchers  at  Dartmouth  College  of  these 
data.  This  analysis  was  able  to  correctly  identify  75%  of  clutter  items  at  threshold.  Although  this 
ROC  curve  looks  substantially  better  than  the  two  above,  it  is  not  as  simple  as  that.  The  first  ranked 
anomaly  list  submitted  by  this  analyst  classified  two  of  the  items  pictured  in  Figure  7-3  as  TOI  but 
had  the  other  four  classified  as  not-TOI  with  one  of  them  late  in  the  ranked  anomaly  Ust. 
Examination  of  the  dig  results  from  this  initial  Ust  alerted  the  analyst  to  the  presence  of  these  items 
and  led  him  to  correctly  classify  these  items  on  subsequent  Hsts. 
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Figure  7-3.  Six  of  the  common  missed  TOI  at  Ft.  Sill 
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Figure  7-4.  Results  of  the  Dartmouth  analysis 
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After  the  initial  round  of  analyses,  both  the  SAIC  and  Parsons  analysts  were  asked  to  re-run  their 
classifier  with  the  knowledge  that  there  were  40-mm  frag  balls  on  the  site.  Both  had  similar  results; 
the  second  analysis  identified  all  the  40-mm  frag  balls  and  added  very  few  extra  clutter  digs.  An 
example  is  shown  in  Figure  7-5.  The  SAIC  analysts  found  the  six  missing  40-mm  frag  balls  with 
only  18  extra  clutter  digs  leaving  only  one  missed  TOI,  an  expended  3.5-in  rocket  motor  with 
significant  damage  shown  in  Figure  7-6.  This  highlights  the  importance  of  performing  a  thorough 
site  characterization  accounting  for  both  intact  munitions  and  their  expected  hazardous  components 
and  communicating  this  information  to  the  analysts. 
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Figure  7-5.  Results  of  the  SAIC  re-analysis  of  the  MetalMapper  data  using  knowledge  of  the 

presence  of  40-mm  frag  balls 


Figure  7-6.  The  remaining  misclassified  item  in  the  SAIC  analysis  shown  in  Figure  7-5 


8.  Conclusions 

Classification  was  used  at  Ft.  Sill  to  successfully  eliminate  about  75%  of  the  clutter.  A  production 
contractor  field  crew  collected  high  quality  cued  MetalMapper  data.  Both  geophysicists  from  the 
production  contractor  and  the  developers  of  classification  methods  were  successful  in  using  these 
data  to  achieve  substantial  classification. 

Ft.  Sill  is  a  challenging  site  for  two  reasons.  First,  the  grids  in,  and  adjacent  to,  the  heavily  used 
targets  had  a  high  anomaly  density  that  necessitated  the  use  of  multi-source  models  which  were  just 
being  perfected  at  the  time  of  this  demonstration.  Second,  the  wide  variety  of  munitions  that  were 
encountered  for  the  first  time  at  this  site  was  a  challenge  for  the  library-based  classification  methods. 

This  last  difficulty  can  be  overcome  through  effective  communication  throughout  the  project.  This 
demonstration  was  carried  out  at  the  same  time  as  the  RI/FS  so  detailed  site  characterization  data 
was  not  available  to  the  analysts;  this  would  not  be  the  case  for  a  normal  project.  Even  in  the 
absence  of  site  characterization  data,  the  class  of  TOI  missed  by  two  of  the  analysts  could  have  been 
correctly  classified  with  better  communication.  All  of  the  UXO  technicians  on  site  knew  to  expect 
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40-mm  balls  on  a  40-mm  range  but  this  knowledge  was  never  transmitted  to  the  analysts.  The 
results  of  every  classification  project  will  be  enhanced  by  transfer  of  all  available  site  characterization 
information  to  the  analyst. 

Finally,  significant  experimental  difficulties  could  have  been  avoided  if  there  had  been  closer 
coordination  between  the  RI/FS  and  this  demonstration.  Using  anomaly  locations  that  result  from 
an  EM61  survey  with  wide  lane  spacing  results  in  relatively  inaccurate  initial  locations  which 
sometimes  require  significant  relocation  of  the  MetalMapper  to  achieve  high-quality  cued  data.  This 
offset  from  the  initial  anomaly  location,  if  not  accounted  for  in  the  intrusive  investigation,  can  lead 
to  recovery  of  items  that  do  not  correspond  to  those  that  were  interrogated  by  the  MetalMapper. 

9.  Acronyms 

BRAG 
ESTCP 
FUDS 
GPS 
HE 
IDA 
IMU 
ISO 
IVS 
LAW 
mV 
QC 
RI/FS 
ROC 
SERDP 
TOI 
UXO 
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